Listing of claims: 



This listing of claims will replace all prior versions, and listings, of claims in the 
application: 

1. (Original) A method of designing an interactive system, the method comprising: 
providing one or more sensors, each sensor adapted to capture one or more real-time 
events; providing one or more perception features, each perception feature adapted to 
receive information concerning one or more captured events from one or more sensors 
and to generate one or more perceptions; providing one or more decision features, each 
decision feature adapted to receive one or more perceptions from one or more perception 
features and to generate one or more decisions; providing one or more output 
mechanisms, each output mechanism receiving one or more decisions from one or more 
decision features, and executing one or more actions; assigning a priority level to each 
perception feature and each decision feature on the basis of a plurality of factors 
comprising duration of an action, complexity of an action, and pre-planning requirements 
of an action; and utilizing said priority levels to load-balance the running of perception 
features and decision features under time constraints and normal system operation, and to 
select among conflicting actions requested by two or more decision features. 

2. (Original) The method of claim 1, wherein one or more of said sensors is a visual sensor. 

3. (Original) The method of claim 1, wherein one or more of said sensors is a spatial sensor. 

4. (Original) The method of claim 1, wherein one or more of said sensors is an auditory 
sensor. 

5. (Original) The method of claim 1, wherein one or more of said sensors is a tactile sensor. 

6. (Original) The method of claim 1, wherein one or more of said sensors is a balance 
sensor. 

7. (Original) The method of claim 1, wherein one or more of said actions comprises 
generating speech. 

8. (Original) The method of claim 1, wherein one or more of said actions comprises 
generating text. 

9. (Original) The method of claim 1, wherein one or more of said actions comprises 
generating phonemes. 
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10. (Original) The method of claim 1, wherein one or more of said actions comprises 
generating an intonation. 

11. (Original) The method of claim 1, wherein one or more of said actions comprises 
generating an animated movement. 

12. (Original) The method of claim 11, wherein the movement includes facial expression. 

13. (Original) The method of claim 11, wherein the movement includes body movement. 

14. (Original) The method of claim 11, wherein the movement includes hand gesture. 

15. (Original) The method of claim 11, wherein the movement includes arm gesture. 

16. (Original) The method of claim 11, wherein the movement includes visemes. 

17. (Original) The method of claim 1, wherein one or more of said actions comprises 
generating physical movement. 

18. (Original) The method of claim 17, wherein the movement includes facial expression. 

19. (Original) The method of claim 17, wherein the movement includes body movement. 

20. (Original) The method of claim 17, wherein the movement includes hand gesture. 

21. (Original) The method of claim 17, wherein the movement includes arm gesture. 

22. (Original) The method of claim 17, wherein the movement includes visemes. 

23. (Original) The method of claim 1, further comprising: selecting one of said decision 
features; and assigning a high priority level to the selected decision feature on the basis of 
one or more criteria, including: a. when output from the selected decision feature results 
in an action requiring less than approximately 500 milliseconds to perform; b. when 
output from the selected decision feature results in a simple action that requires few 
motors to perform; c. when the input to the said decision feature are output produced by a 
perception feature or features with a high priority; d. when the inputs to the decision 
feature come from a relatively few perception features; e. when the decision feature 
requires relatively simple processing; f. when the amount of preplanning required for the 
decision is low or none. 

24. (Original) The method of claim 1, further comprising: selecting one of said decision 
features; and assigning a medium priority level to the selected decision feature on the 
basis of one or more criteria, including: a. when the selected decision feature results in an 
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action requiring approximately 500-2000 milliseconds to perform; b. when the selected 
decision feature receives output from a relatively large number of features; c. when the 
selected decision feature is clearly related to hand-holding of the interaction between the 
system and a system user; d. when the input to the said selected decision feature is from a 
perception feature with a high or medium priority level; e. when the amount of 
preplanning required for the decision is intermediate; f. when the action resulting from the 
decision feature's decision output is complex, or requires a high number of motors to be 
performed. 

25. (Original) The method of claim 1, further comprising: selecting one of said decision 
features; and assigning a low priority level to the selected decision feature on the basis of 
one or more criteria, including: a. when the duration of an action resulting from said 
selected decision feature is approximately longer than 2000 milliseconds; b. when said 
selected decision feature requires complex processing; c. when said selected decision 
feature requires large amounts of pre-planning; d. when said selected decision feature is 
related to a knowledge-based topic other than the interaction itself; e. when the input to 
said selected decision feature is from a feature with a medium or low priority level. 

26. (Original) The method of claim 1, further comprising: selecting one of said perception 
features; and assigning a high priority level to the selected perception feature on the basis 
of one or more criteria, including: a. when input of said selected perception feature is a 
single data entry, or a collection of very few data entries; b. when input of said selected 
perception feature comes from a single sensor; c. when the processing time of said 
selected perception feature supports one or more actions having a perception-to-action 
loop time that is shorter than 500 milliseconds; d. if one or more decision features 
supported by said selected perception feature supports one or more outputs that result in 
an action requiring less than approximately 500 milliseconds to perform; e. when data 
needed already exists as raw data; f. when data gathering takes roughly 300 milliseconds 
or less; g. when input of said selected perception feature is produced by one or more 
perception features each of which have been assigned a high priority. 

27. (Original) The method of claim 1, further comprising: selecting one of said perception 
features; and assigning a medium priority level to the selected perception feature on the 
basis of one or more criteria, including: a. when data gathering of said selected perception 
feature spans approximately between 300 and 1500 milliseconds; b. when input to said 
selected perception feature comprises a relatively large number of data points from more 
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than one sensor; c. when a decision feature supported by the output of said selected 
perception feature is directly related to an interaction between said interactive system and 
its users; d. when data processing of said selected perception feature requires information 
from various parts of said interactive system, including the context of an interaction; e. 
when said selected perception feature supports perception-to-action loop time between 
500 and 1500 milliseconds; f. when said selected perception feature takes between 300 
and 1500 milliseconds to produce output; g. when said selected perception feature's input 
is the output of other perception features that have been assigned a medium or high 
priority level; h. when said selected perception feature's output supports a decision feature 
whose resulting action takes between 500 and 1500 milliseconds to perform. 

28. (Original) The method of claim 1, further comprising: selecting one of said perception 
features; and assigning a low priority level to the selected perception feature based on one 
or more criteria, including: a. when data gathering of said selected perception feature 
requires longer than approximately 1500 milliseconds; b. when data processing of said 
selected perception feature requires complex data or knowledge structures for producing 
output; c. when said selected perception feature produces complex, knowledge-based 
output; d. when said selected perception feature's output supports processes for high-level 
decision making such as complex natural language processing or planning ahead; e. when 
data processing of said selected perception feature takes more than 1500 milliseconds to 
perform; f. when data processing of said selected perception feature requires information 
from multiple parts of said interactive system, including the context of an interaction; g. 
when said selected perception feature's input is the output of other perception features that 
have been assigned a low or medium priority level; h. when said selected perception 
feature's output supports a decision feature whose resulting action takes longer than 1500 
milliseconds to perform; i. when said perception feature is difficult or impossible to 
decompose into small perceptual features; j. when said perception feature typically cannot 
support a perception-to- action loop of shorter than 1500 millisecond duration. 

29. (Original) The method of claim 1, wherein the method is used to design an interactive 
system that implements one or more anytime algorithms. 

30. (Original) The method of claim 1, wherein the method is used to design an interactive 
system that is implemented in a distributed, asynchronous architecture. 
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